It has long been the case that it has been possible to query databases holding data to provide a response to the query. Queries are run on databases to find a match for the information being requested responsive to the query. For example, a user providing the query might want to know how many entries in a particular database satisfy a particular requirement, for example, an age range or gender requirement. There are numerous technologies available to handle this.
Techniques for using more than one database to satisfy a single query having multiple expressions are known. According to one technique, two independent datasets are firstly merged together so that they exist as a single dataset which can then be queried with the multiple expressions. This procedure can require the transfer of large amounts of data, and a complex merging exercise at the receiving end.
Nowadays every company holds valuable data, for example concerning attributes of consumers or potential consumers who may wish to access that data. Sharing customer data among different companies creates complex privacy and legal issues, as the data contained in customer records may be shared involuntarily. The problem is exacerbated by the fact that different departments within the same company, or among different companies, may store data at different locations, and thus moving and selling data across sites poses security risks that may result in value leakage. Furthermore, different companies' storage formats are often not compatible and therefore increases the complexity in sharing customer data.
In the United Kingdom, the general data protection regulation defines a number of important principles around the storage of data. These principles include data minimisation, storage limitation and purpose limitation. For the purpose of data minimisation, the regulation states that personal data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed. For storage limitation, the regulation states that personal data shall be kept in a form which permit identification of data subjects for no longer than is necessary for the purpose of which the personal data are processed. This regulation and other similar regulations in other countries have placed further constraints on the storage and access of data. This is in direct conflict with increasing requirements of companies to utilise data for the purpose of analysing and gaining insights.